In Mixed Reality (MR), virtual and real worlds are combined to obtain a seamless environment where the user interacts with both synthetic and physical objects. In mixed reality, an individual must believe that the real and virtual worlds are one and the same. While MR is defined to span from the purely virtual to real-world experiences, most MR environments are in one of two categories: augmented reality where virtual elements are inserted into the physical space or augmented virtuality, where real elements are composited into a virtual scene. A technology that enables this full spectrum of MR is the video see-through head-mounted display (HMD), a wearable device that captures the user's view of the environment through mounted cameras and displays the processed images to the user via screens in from of the eyes. The HMD is especially useful in augmented virtuality where real objects must be separated from the regions where virtual objects will appear in the synthesized image. Typically, blue or green colored physical objects can be used to identify the virtual surround, thereby making chroma keying an effective technique for defining this matte.
When using HMDs, the cameras transmit fairly low resolution video streams with high signal noise. The multiple streams must be processed in real-time for an interactive user experience. However, matting is under-constrained and ill-conditioned due to limited spatial resolution and a large amount of signal noise. Commercial hardware, such as Ultimane®, produces high fidelity mattes by using heuristically-based chroma keys for a single video stream. These hardware solutions are costly, especially when considering multiple users with multiple cameras per HMD. One technology that enables this perception is the video see-through head mounted display (VST-HMD). Another technology that aids in the merging of both environments is chroma key matting. A common downfall of the VST-HMD is that low resolution and high pixel noise cause difficulties with chroma keying solutions.
Chroma keying, often called blue screening, refers to classifying a range of colors, commonly in the blue or green color spectrum, from a captured image to define a matte for compositing multiple image sources. The problem is under-constrained and most commonly addressed through commercial products that require tweaked parameters and specialized hardware for real-lime processing. In Mixed Reality (MR) applications using video see-through head-mounted display (HMD) hardware, the problem becomes more difficult since the poor camera quality within the HMDs, interactive processing requirements of MR, and multiple camera sources demand a robust, fast, and affordable solution.
A number of patents exist which relate to chromakey matting, including, U.S. Pat. Nos. 7,508,455, 7,006,155, 6,927,803, 6,897,984, 6,807,296, 6,501,512, 6,437,782; all of which are incorporated herein by reference.
Prior techniques require low signal noise, significant calibration or do not run in real-time.
Hence, there is a need for an affordable, robust, and fast matting process that works with low-quality video signals, less than perfect lighting environments and non-uniform matting material (e.g., low-cost, light weight cloth rather than rigid surrounds).
The present invention is designed to address these needs.